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ABSTRACT ^ J 

This document repor**,the pilot test*of the two 
components of the, Child and Family Mental Health (CFMH) Evaluation 
Project — the impact evaluation component and the in-dep"th 
evaluation component . (The impact evaluation * is designed to determine 
the effects of the two primary prevention models of service and 

^activities on the CFMH Head -St*rt programs 'a? compared to t^eir 
designated -control^. The in-dtopth evalixat ion^is designed to assess 
the effects of the CFMH's r^Jwt^ prevent i ve activities on Head Start 
children, families ,, staff , aW^center atmosphere.) First, the 
document specifics the procedures to be usfcdtin the pilot study for 
selecting programs and sample's of Head Start children, teachers, and 
' parents, outlines observational schedules, and discusses procedures 
for the recruitment, hiring, and training of tfltaff . Next, the 
document btiefly reviews - the instruments selected for the evaluation 
and ekplicates the conditions of us$, revisions , ^and permissions 
attained to use the instruments'. Finally/ the /site monitors 1 field ^ 

'operations ^Oi data management procedures are described. / 
(Author /MP) J 
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' The Spring data collection site visits were designed to obtain 
process evaluative data and to pilot test both the measures and proce- 
dures to be* used in the impact ,and in-depth evaluations in Phase III. 
The process evaluative data were to' have been collected in all €xperi- 
mental and Control sites as the post-test for Phase II. At the time data 
collection was scheduled to begin* the Office Management and Bucfget (OMB) 

had not granted approval to use the pitfcess instruments previously design- 

9 x 

ed and field tested. The collection of process^data was postponed and 
subsequently cancelled after approxiinately two^weeks of a five Vfefck data 
collection schedule had passed^ ^ 

> The pilot studies o£ the impact and in-depth evaluation proceed- 

ed as the previously published measures to be used did not require 0MB 
approval. Procedurally, the major ramification of cancelling the process 
data collection was the loss of an opportunity to assegi the impact of 
collecting process, impact and in-depth data on a single site visit as 
required in Phase III of the evaluation. 



The impact evaluation is designed to determine the effects of 
the tw0 primary prevention models .pf service and Activities in the CFMH 
Head Start programs 'as compared to their designated controls,. The in- 
depth evaluation is designed to be a giver-tuned assessment of primary 
preventive activities 'on Head Start children, fajnilies, staff and center 
atmosphere. The in-depth evaluation focuses on a subsample oi the CFMH 
Centers and will emphasize observations made by a third party. The in- 
deptluevaluation complements the impact evaluation by providing more 
intense^'and precise assessments of program impact 

N The contract scope of work stipulates that the {Jilot study of 
iiipact measures be performed at a minimum of four Head Start sites. Two 

' * of the sites had to be experimental programs and two had to be control 
programs. Of . the two experimental programs, one was to be a Mental 

t Health Worker Model and the 9<her a Community Resource model. The con- 
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tract also specified that the pilot testing of in-depth nteasures' wc!re to 

be conducted in two programs — one an experimental and one a program. . 

These stipulations defined the parameters within 'which site selections 
i 

. were made. V « 

< « S 

Selection of Pilot Programs . 

si * 

Within the parameters explicated by the contract, there were 

several additional considerations influencing the selection process. To 

* - * * 

be maximally beneficial, it -was j^ged th^t the programs selected for ■ ' 

the impact ana in-dept*h pilot studies should be characterized by: ' (a) the 

• ■ * 

use. of a range of primary preventive activities in order to test the 

sensitivity of 'the measures tb reflect the ^impact of a variety of inter- 
vention strategies; (b) "the implementation of primary preventive acti- 
vities which offer some promise, of being &ffective^ and (c) the use of 
primary activities that were generally' representative of the range o£ 
experimental and control programs. It was furthai: decided to select at 
least. one Head Start .program with multiple centers some distance from 
each other. Such a center would allow the field procedures to be tested 
by the logistical ch alleng es offered by geographically 'dispersed centers. 
Finally, cost cons idJBBp^ugges ted that we select Head Start progfams 
in close t proximity for the pflot test*. Therefore, dcfrisiderations of 
travel cost were given priority over any advantages offered by the use 
of sites yoked as experimental and .control. -With these constraints in 
ipind, the selection process -progressed through several stages. 

The CFMH ^xperimental programs were categorized into .Mental 
Health Worker and* Community Rasourc£ models . Descriptive information on 
^>each Experimental program was compiled from program- grant proposals, : 
the Urban Institute fox Human" Services 1 T>hasei I Report, and information 
gathered during the Fall 1979 site visit* (i.e., site monitor reports 
and process interviews with the Mental Health 'Providers) . From these 
source* was abstracted information- on the size of theprogram (e.g\ , 
nunrtrer b,f children enrolled, the number of^centers)? the administrative. 
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functioning of the program (e.g., extenttfbf staff turnover and internal 
politics); the activity of the CFMH project (extent of parent participa- 
tion, the types of activities); and practical logistical ifiEormation 
(availability of s^ace for interviewing teachers and parents, amount of 
- class time when children aty teachers could be observed, -quality of 
local interviewers, cooperation of parents, extentr^f perceived physical 
danger to outsiders)* v 

• • 

Experimental -programs ' were eliminated from consideration as a 
pilot site if this information indica^ted: a relatively inactive CFMH 
^project; significant Head Start personnel problems; perfeived physical 
danger to Urban Institute for Human Services 1 field persons; or non- 
representativeness of Head* Start program (i.e., very large of very small 
.program, completely urban program, 'unique relations with other agencies 
such as a public school system or a' migrant program). As intended, only 
a few programs remained from which to select one Mental Health Worker 
model and one Community Resource model. Since any of these few programs 
were appropriate choices, a quasi-random selection wa^ madel The Commu- 
nity Resource Experimental program selected Indiana, Pennsylvania, and 
the Mental Health Worker model program selected, was Georgetown, Texas. 

v 

The ACYF Regional offices wdre contacted to determine 
selected programs were 4-n compliance with the Head Start mental health 
'guidelines. The Support Services Contractors were also contacted to 
discern whether they had information that would indicate that these 
yere inappropriate choices. The two programs, Indiana, Pennsylvania and 
Georgetown, Texas were readily acceptable to the Support Services Con- 
tractors and the programs 1 mental health components were in compliance 
with federal guidelines. t ' ' ' 
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Sample Selection 

Th^ impact and in-depth pilot tests required that samples of n> 
Ulead Start children, teachers, an^ parents be selected. The contract 

stipulated sample sizes of 20 and 40 children for the impact $nd in-depth 
. pilots, respectively. It was decided to mak,e the child the sampling unit 

and link the parent to the child .selected. Therefore, the child and 

parents' samples were selected in one procedure. 

The size 6^ the population from which the child sample was 
selected was reduced by just selecting centers and classrooms from which Jfe 
the sample would be drawn. Centers within Head Start programs were 
sfelected to reduce the amount of travel necessary to observe all children 



in 



the sample. Within these centers, *3.assrooms were selected in a. 



manlier that allowed child observers the option of moving to another class- 
room to observe othter children within the same day. 



Within* a sample classroom, letters were 'Sent to parents to 

obtain written permiMion to allow their chil^ to participate as well 

as to obtain their consent to be interviewed. The actual selection of 

the child was made by the child observer after entering the selected 

classroom. From a list of children whose parents had provided' written 

consent and who were present, the child observer selected' from 4 to 5 

children equally divided between girls anci bo^. Thus, a ^hild became 

a pajt of the sample only if ,heV she was observed. 

S 

i -Teachers and teacher aides were included In the sample only 



if their classroom was selected and children in the classroom were , 
selected in child samples. The' number of tea.chers to be included in 
the sample was not Specified in the contract. {A decision was made to 
maximize the number in order to maximize the set of data on which fujture 
evaluative an<J, policy decisions are+ased and to decrease the time any 
individual teacher* mqpt ddvote to rating the children in his/her class. 
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^Observational Schedules 

The number "of^Jays required to complete data collection on 
each site was partially determined by 'sample size, the amount of time- 
required to complete each measure, the numbeT of observations to be 
made, and, the availability of key Head Start personnel. These factors, 
iti turn, determined the length of the number of days on each site. 
Since the observations were' the most costly measures used, the schedule 
was designed to maximize the probability that the, required number of 
observations were made.. 

All children were to be observed and their behavior cqded on 
two- different accasions. Since one child observation required approxi- 
mately one hour, two observers would require four to five mornings to 
complete the required 20 sets of observations for' the impact evaluation. 
Therefore, the impact site visits were scheduled for one-week. The 
in-depth samples of 40 children required a two-week schedule, Schedules 
of observations were prepared to- allow each observer to be in each* qlass 
room once in four or f\ve days. . Observations were scheduled to allow 
50% of the child and teacher observations to be made simultaneously and 
50% to b^ made at different times. The procedure was designed to * 
determine the differential effect of having one-versus-two observers in 
the classroom. The pairing of child and , teacher observers was rotated s 
that each child obsetver and each teacher observer was matched at least 
once during a week of . observations . A sample schedule of child and 
teacher observation is attached as Appendix A. The optimal .observation 
schedule was modified in practice to adjust fot absence/of cl\i/ld or 
teacher on scheduled observation days, and scheduling difficulties. 

Recruitment and Hiring of Field Staff ' / 

The impact and in-depth evaluations required three types of 

— 

field staffs A kite monitor was needed to direct the data collection 
* si 

teams at each Head Start program. Observers were needed to make child 

* ~ 
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and teacher observations as well as interview Head Start teachers 
Interviewers, who* were residents of the cities in which the Head 
Start programs were located, were needed to^ interview Head Start* 
patents.*. ( 



•Recruitment for Site Monitors and observers involved the 
circulation of job announcements to^ the colleges and uivi.versj.ties in the 
Bay Arda and the Calif ornife Department ^of Employment.. As well, an 
Advertisement- was placed in the^ employment sectidn of the San Francisco 
newspapers. A further search for Spanishrspeakinj applicants for the 
observer positions was* conducted by contacting employment counselors, 
selected faculty members of Bay Area colleges an^-Qniversities , and by 
soliciting the aid of directors of La Raza^ and etTkjic studies programs. 



The applications, resulting from the job announcements were 
^ reviewed and screened by the research scientist in charge and ranked on 
the basis of # general research, interviewing, classroom and observation 
experience. Selected applicants were invited to be interviewed." During 
✓^€toe~tfiterview, the specific requirements of the positions were explained. 
In the case of appli'cants^f or observers positions, it was emphasizSti 
that observers would only be sent to the field if he/she achieved an 
80% reliability secure on the observational measure on which he/she would 
be trained. < ' 

i < 

t The outcome of the process led to the hiring of nine' Site 

f * Monitors (sfeven for the field teams 4nd two alteuaates^anl ten observers 
(4 field team, child observers and 1 alternate; 4 field team teacher 
observers and*l alternate). The alternates were hired for the training 
' period only. Their assignment to the field team was contingent upon a 
vacancy becoming available. 
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Training of Site Monitors 



The ttainir\g of Site ltonitors was. divided into two sessions. 
Three- dsiys of trainings took plrce prior to the site development visits 
(March 12-14, T980) an additional three day session (April 7-9, 1980) 
was conducted after the site development visits* but before the data 
collection visiTs.. Copies of the training schedules are reflected on 
the following, pages. ^The thrust of the .training was to: (1) provide 
Site Monitors wl£h the philosophy and programs of Head Start; (2) pro- 
vide a detailed description' of the CFMH program; (3) provide a complete 
understanding of the CFMH "Evaluation Project; and (4) train Site Monitors 
on the use of data* collection instruments. The second training session 

included a debriefing discussion on site development, training on the 

'i * • i 

N use of the CFMH log, a session on training interviewers to administer ^ 

• parent interviews, a refresher training on interviewing techniques, and 

a session on administrative procedures. 9 . 



Training of Observers 



The training of Child and Teacher Observers was conducted in 
two parts. The first part of the training was conducted in two parts. 
Th£ first part of the training was conducted by SRI-International. 
During this part of the, training the Child and Teacher Observers were 
'trained separately. The agenda and format of the training sessions yere 
very Similar. In botf/ cases, trainees needed to become familiar with 
the physical f ormat ^f^hpcoding system, the code names ^f or the behavior 
categories, the contents of each category and its boundaries. 

The second part of the training was conducted by the staff of 
the Urban Institute for Human Services. On the last day, training of 
observers and Site Monitors was combined. A detailed description of the' 
observer yraining follows. 
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t Trailing by SRI-International. One week before a training 

session started, the Urban Institute mailed a training packet^(provided 
by SRI-International) to each observer. This packet introduced the 
behavior codes used by the observation system, with definitions and 
■example behaviors.. Trainees were tfeld to know the oodes by thk^lrst 
training session', using the exercise sheets included with the pafcket , 
to facilitate learning. The training, agenda was briefly outlined (see 
Appendix! B)>. 



Midafternoon on the fourtfi day, all observers and Site Monitors 
ame to the Urban Institute to sign^ contracts , their contracts to, ( 
receive pay checks and advances. The Impact and In-Depth Research 
Scientist met briefly with each observer individually to congratulate 
the person on hif/her fine performance during 'the training, to entourage 
hjm/her to work GloseLy *wittv the Site Monitor ancj other observers, to be 
sensitive to Head Start relations ant^ask if there were «ny individual 
problems wHich the Research Scientist should anticipate and any other 
phone n\imbers the Research Scientist should have. 

The fi*st two- days of the training sessions were spent i*^te&c- 
ing and alsp ^test^ing the trainees' knowledge of the codes. Fla'shcardl 
were used to erfourage speed in encoding. Coding practice was given, 
using videota^d vignettes of nursery school interactions. Each evening 
there wc/re' homewor^ problems wfcich were discussed the following day. § 
The third through fifth days, trainees spent two hours at a local nursery 
school, observing, and coding the behavior of the focus person. Discus- 
sion of coding problems followed each observation session. Videotaped 
vignettes were also coded and cussed. Practice reliability- testing 
start^tT on tHp second .day and occurred each day thereafter. The sixth 
and seventh days were spent solely at SRI. '^The final day consisted 
mainly of warm-up coding and then coding of criterion tapes to establish 
a reliability score for each trainee. Feedback was provided after 
scoring w#s completed by the trainer and tfie Research Scientist. 

r 
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'All trainees achieved better than the requisite 80% agreemenjf^ 
with -the trainer's coding* The five Teacher Observers (SRI Preschool 
Observation Instruments, scored Preschoo'l Observation Instrument) scored 
92* $0, 88, 88 and 86; the, average^aliability SQore was 88,8%, The 
Child Observers (Prescott-SRI Child Observation Instrument) scored 94, 
'93, 91, 90 and 80; the average reliability score was 89.6% agreement 
with the trailers. 

/ 

Onevor two weeks latep, and 2-3 days before entering the 

field, the observers returned to SRI-Int;ertiational for 1 0ay of refresh- 

er training, ' During this session, observers coded -videotape vignettes, 
» 

discussed problems 'arid took a shorter reliability test. Again, all 
observers achieved better than the requisite 80% agreement with the 
trailer. Teacher Observers displayed an average score of 91.5% with 
a range of '90-94. Child Observer scores ranged from 80-100 with ap 
average of 93.3%. 

I' 

Observef training by. Urban Institute staff. This training 
session lasted three days^and was the first time the tw^field teams 
met and worked together.* A rapid pace was set and maintained because 
there were so many topics to be introduced and assimilated. 

To facilitate the assimilation process, observers receiypd a . 
second training packet two weeks before the Urban Institute training 
session. This consisted of a complete spt of the teacher measures 
(i.e., the CFMH Process Interview, the CIRCUS Educational .Environment 
Questionnaire, the Kohn Social Competence Rating scale). The appro- 
priate observers also received the Brown IDS Self-Concept Referencje 
Test. Observers were asked to familiarize themselves with each measure ' 
|and practice administering it (see Appendix C) . 

On the first day of training, observers met separately from 
the Site ^Monitor-s (Process-only, Impact and In-Depth, and laternates). 

After a brief introduction by the Research Scientist and the CFMH Pro- 

/ 
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• . ject Director the Observers were welcomed and presented -an overview of' 
the CFMH Project. The Research Scientist then delineated the *tasks to 
be accomplished in the following three dayp and emphasized that the * 4 

training meetings were to be viewed as seminars, aot classes (i.e., ^ 

( it was important *that, everyone -participate^ by asking questions, raising 

problems, and of f ering % solution&) . < The Impact and In-Depth Site Monitors . 
as well *as an alternate Site Monitor wefe briefly introduced to the 

i Observers '. The Impact and In-Depth measures were reviewed; problems • * 
that had arisen in administering them were discussed. At this time, p - 0 

Observers were informed that the Schedule of Recent Experiences had been 
dropped f£om the In-Depth battery of measures. - * 

# 

Observers .received their training manuJ^^ and thei*r responsi- 
bilities and taska^rere received for questions. Topics discussed were 
, general responsibilities, introductory meetings with Head Start staff, 
child observations andteacher observations in classrooms, teacher inter- 
views, teacher aiide meetings, ctfild interview, editing of observation ' 
booklets apd interviews, "daily meetings with Site Monitor," final Observer 
and Site v Monitor meeting, -'on-site^ sampling procedures and the Observer ' ' 
/ report (see Appendix D for greater .detail) . * The final task of the first 
day was to k p^rfoim Dractice observations using yideotapes borrowed from 
SRI-Internatio^al. ^This.task was included .because SRI experience has 
shown that Observers ^need to practice the codes ^dail^ to- retain facility 
in translating observations into codes. The higher reliability scores, 
obt^ined^uring the SRI refresher training indicate the usefulness of 
this ad/fitiofial practice. An audio tape was : toade of this .training ses- 
sion. • . ^ 

The second da^of Observer training at the Urban Institute for ^ 
Human Services was conducted / ^intly with Sitfe Monitors * After a 'brief 
introduction of all participants, the morning session began with training ' 
on interviewing techniques an<kthen practice interviewing, using vthe 

Teacher-Process instrument. ' (At th^time, OMB clea^nce of this and \ /" 

other Process instruments was anticipated.) Responsibility for the after- 
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noon session was shared among* the Urban Institute staff. The Impact and 
In-Depth Research Scientist introduced thelrole of the Observers to the , * 
Process Site Monitors and briefly discussed the role-relatiwsh'ips among 
.Site Monitors and Observers. Questions were ans^red. The Process 
Research Scientist discussed the entrance and exit meetings to be. held 
by thfe Site Monitors at each ptogram. 'The Research Associate discussed ■ 
administrative and logistical information, concerning consultant fee' 
claims, expense records, itinerary, car rental, and the identification 
numbering system (for Head* Start prog-rams, center and classrooms,' respon- 
dents, Sbseryers and interviewers). Plane ticketsrwere distributed. 
Many questions were posed and answered* .Finally, all Observers and Site 
Monitors signed an affidavit of confidentiality. 

On the third and fourth, days of training, half of the Observers | 
attended the SRI-ref resher training session, while the other half attended 
the Urban^Institute training. The Impact and In-Deptfh Site Monitors 
attended these last days of Urban Institute training. The' Impact Site 
Monitpr reviewed. and expanded upon the inteyview training provided the 
previous day. Specific questions were discussed and answers provided. 
Practice observations' using, videotapes Were made. In the afternoon, 
Observers ^received the Observation Schedules for the first week of- data * 
Collection, these were discussed ^and questions were answered. The In- 
Depth Child Observers practiced administering the Brown Self-Concept 

* 

Test, including the operation of the Pplaroid cameras. 
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i*. { Impact' and In-Depth Measures 

The procedure ind rationale used for the selection/of Impact 

and In-Depth measures are presented in a previous report entitled Rlviev 

and Recommendation of Impact and In-Depth Instruqertfs . The following 

section briefly reviews the instruments selected and explicates the 

K - f 

conditions -of use, revisions, and ^permiss ions attained to use the instru- 

* 

ments.. Al impact and in-depth instruments were previously published. 
Impact Measures 

* 

Two observation measures were selected, the SRI-Prescott Child 

Observation/ System and the SRI Preschool Observation System (Adult Focus) 

I* 

Both these measures are closed, technical systemj^wh^eh require highly 
trained observers for implementation. A second teacher measure is a 
portion of the CIRCUS 17 - E^u^tion^l Environment ^Questionnaire . The 
measure selected to assess the impact of the CFMH Project on parents is 
the Home Inventory Scale. 

The measure of impact of the CFMH Project on the child in addi- 
tion to the Prescott, Observation System, is the Kohn Social Competence' 
Scale. This is scale, to be completed by a child^s teacher and also by a 
teacher aide. A revised version of the Kohn Social Competency Scale ; was 
prepared to provide a parent 1 s rat ifig of her/his child. 



SRI Observation Instruments . The SRI -Prescott Chil^ Observation 
System and the SRI Preschool Observation System (Adult Focus) require 
specially trained observers, specially-developed optically scannable 
booklets for recording th^ data and specific miscellaneous materials 
such as pens, signal boxes and earphones. Permission to use the two 
observation systems required earphones. Permission to use the two obser- 
vation systems required contracting with SIU to provide observer training 
'sessions at SRI in Menlo Park, California* consisting of 7 full days of 
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training for 6 (or fewer) persons and 1 day of refresher training for 
each system. ' ^ 

SRI' requires tfiat>^» Observer trainee attain an 80% relia- 
Mlity rating before SRI yf-11 recommend that person as an Observer/ In 
the past / some trainees have not. Achieved that level of expertise by the 
* , end of the training period, m 

w 

In addition/ SRI agreed to provide the Urban Institute for 
Human Services°with marking pens, 250 SRI Preschool and 90 SRI-Prescott 
observation booklets, and to rent the necessary signal boxes' and' beepers • 
The Urban Institute for Human Services vas to be responsive for paying 
Observers' salaries and expenses during the training sessions*. SRI 
agreed to make recommendations to Urban Institute for Human Services 
regarding the capability of each observer to collect reliable data. The^ 
Urban Institute for Human Services was to be responsible for editing, 
scanning, and processing of the resulting data, and issued a fixed-price 
purchase order to S&I accepting these conditions. * 

SRI was able to provide a sufficient quantity of Preschool 
observation booklets to cover Urban Institute needs, but they did not 
have sufficient Prescott booklets. NCS, Minneapolis, possesses the 
negatives for printing the booklets, but the price for printing the small 
-quantity needed for the Urban Institute pilot ^est appeared prohibitive. 
Fortunately, ^j)p lied Management Sciences, Silver Springs, Maryland, 
possessed 50 extra Prescott booklets, which they gave to the Urban Insti- 
tute for Human Services. , * m 

CIRCUS 17 - Educational Environment Questionnaire . Urban 
Institute received permission from Educational Teating Service, Princeton, 
New Jersey tp produce and* administer 50 copies of portions of CIRCUS 17 
^nstrument via a licensing agreement . The Urban 'institute agreed to pay 
a. nominal licensing fee to have each copy bear the, notation: 



J 
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Copyright 1972, 1974 by Educational Testing 
Services/ Xll rights reserved. Adapted and 
. reprinted^ ynder license^- and to provide ETS 
. with a, copy t>f each variant edition. 7 

Home Inventory Scale . The Urban Institute received written 
permission to reprint^Opies^jf the icale from the High/Scope Educational 
Research Foundation. Tjfie Orban Institute was asked po print the Founda- 
tion name and address \pf he copies, and in any reports.^" ThePUrban Insti 
tute translated the sca^fe^mnto Spanish. 

* Kohn Social Comperenc& Scale . This scale was developed for 

teachexs of preschool children. Bec^use^the Urban Institute desired-^) 
parent ratings of a child's social coHlpetence, in addition to teacher 
ratings, we requested permission, toj adapt tfa£^ scale, as well as administe 
*the classroora^scale. Dr. Martin Ko^in granted permission to do both and 
offered to .comment on the adapted s^ale. In a phone conversation, he 
made severa^useful suggestions and approved the adapted version. ^ The 
Urban Institute for Human S^tvices typed arfcl reproduced the necessary 
quantities ^or both scales* The Kohn Social' Competence Scale for Parents 
was translated into Spanish. 

A ■ ' 

In-Depth Meafaires- t « . . ' - 



The in-depth Measures included portions of the Parent Attitude 
Inquiry and the frown IDS Self-Concept Referents Test. The schedule of 
Recent Experiences (SEE) was dropped from the proposed battery because 
of the heavy response load required by the combination of other measures. 
The SRE was originally 'selected to provide "an assessment of the level of 
stress in the lives of parents and teachers. 



^igh/Scope Educational Research Foundation, 600 N. River St, 
Ypslanti', Michifcaft 4B197,, (313)485-2000. 
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Parent Attitude Inquiry- This Was 'developed as an ad hoc 
measure far use with college-educated' parents, as part off' a broader study, 
of patterns of parent authority, by Dr. Diaftia Baum^ind, University of 
California,. Berkeley. 2 Ttie Urban Institute requested permission. to i admin- 
ister a* subset of the items and to revise some of th£ working; this ptermis- 
sion was granted. 



4 



¥ BrowV Self-Congest Referents Test . Nonexclusive and royalty- 

free 1 'pe«mission to administer- this test>as granted in writing by Educa- 
tional Testing Ser^iice. ETS stipulated that , copies carry the statemeirt 

"Reprinted by permission. Developed by B.R. Brown, Rutgers University. 

»> ^ 
Dr. Brown also gave, verbal permission' to administer his tgst. The Urban 

4 W 

Institute reproduced the necessary quantities, and also produced a 
Sp^ijish translation ot^ the instrument. 

A first task in preparing the measures was to satisfy the con- 
ditions imposed by the developer; generally, this was to identify the 
developer on the measure itself. A second task was to obtain Spanish 
translations of the parent and child* interview measures. -This task was 
subcontracted. All Spanish interview schedules were color coded. 

The order bf administration of the parent and teaser measures 
was ctefermined. For the parent interview, it was decided to administer 
the High/Scope Home .Environment Questionnaire*^ irst. This is the least 
judgmental of the measures, asking about specific playthings available 
to the child and how often the parent and child do certain activities; it 
is also krief. .The Kohn Social Competence Rating Scale was ordered second, 
and before th? Process interview, to serve as thought-provoking lead-in' 
into t*e Process instrument. The phn questions about specific types of 
behaviors and how often the child performs them. (Because OMB clearance 



2 See Baumrijid, Diatva, Current Patterns of Parental Authority, 
Developmental Psychology Monograph , 1971, Part 2). 
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was not granted by the time, of the Spring data collection period, the. 
Kohn measure did not serve that purpose.) For the In-Depth interviews, 
the Parent Adfitude Inquiry was ordered last. Respondents rarely enjoy 
a long atti/ude inquiry, so it was ^deemed wise to pi;e£ent ithe post annoy- 
ing measure last. m ' * 

T^e 'teacher measures were organized as follows: The 'CIRCUS 
Educational Environment Questionnaire, two or three Kohn Social Compe- 
tence Rating Scales, the Spring Process interview, then the remaining^ 
Kohns. Because the Process instrument was not administered, it could 
not jferve tKe function ofr a respite from rating children., 
« • ^ 

All measures, typed by t\xe Urban, Institute or duplicated from 
develpper/ s copy, had the spaces for name and school deleted from the 
copy. Interview booklet coversWere designed'^ display "the' necessary 
identifying information (Head Start program ID number Center/Classroom 
(ID //, Respondent /Interviewer if, date -of interview and length if interview 
in minutes). Covers >jp:e titled Parent Intervi^r, v Teacher Interview, 
,Kohn^ocial Competence Scale for Teachers, Brown IDS Self-Concept Refer- 
ent^ Test. Impact^or -In-Depth was stamped in large block lettei^ across 
t(ie top of each cover. The covers of interview booklets w^re color- 
co'defl. It was determined that tl?e impact and in-depth measures should be 
conducted verbally, so as not to assume literacy. This necessitated the 
use qf a cue sheet- with, the Kohn Social Competence Scale. The cue sheet 
provided the five^hoices of answer: HARDLY EVER QR NEVER; SELDOM; 
SOMfiTIM^S; OFTEN; VERY OFTEN 6R ALWAYS. Special translations were made 
of all parent and child interview measures. 



fielC oper/Ct^ons • ' 
k 

Field operations technically b£gan with the initial contact 
with the Head' Start Directors. The (majority of early contacts with the- 



pilot sites were conducted through letters, samples of which are included 
in the^ppendix The first contact- regarding the pilot' test was ijiclud- 



ie^pp 

ed wiJra letter 'informing each program of ttii upcoming site development 
and data collection visits. Tentative dates were scheduled and the 
Director was asked to agrie to it or infturta the Urban Institute that the 
date was impossible, \ second letter infon^d the four pilot sites that 
they had been selected^ "The letter included a brief description-of the 
Impact and In-Depth measures, who , would be observed and intetyiewed, and 
how long^the data collection visit! would last. Dates were settled upon 
by all forfr programs . K . 

> • * / " . ■ : 

Site Development ^ 

The next Urban Institute contact with the pilot programs was 
the two -day site development visits by the Site Monitors. As mentioned 
in the previous section on Site Monitor training, the Site Monitors 

had several tasks to perform during the site development visits. The . 

* 

Impact and In-Depth task% were to acquire (or develop) class rosters of 
the Sample classrooms and o'bttin £h€^parent names and addresses. The 
Site Monitor was* to determine which children were Fall Process children 
an*d Jj>clude them in the list to whom letters tffere sent.' More children 
were to be randomly, selected front the rosters to achieve a list of 40 
(Impact) or 60 (In-Depth). The Site Monitor wpa- to send letters to these 
parents requesting permission to observe (and \int;erview) the child and 
review the child's records Jsee Appendix E) . The permission slip was to 
be sigaed and reu|rneid to the program. This task was accomplished with 
varying degrees of "difficulty . Three of the £dar pilot programs easily 
provided class lists and parent addresses; the fourth program did not** 
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have class rosters available in the central office, so the Site Monitor 
spent several hours' .asseiftbling a partial list. Arrangements were made 
for the Research Scientist to call t?he programs periodically to le&rn 
Che status of the returns of parent permission slips. 

s 

v 

A second ^ask was to arrange/appointments with Head Start staff, 
inserter as possible, for, entrance meetings and interviews. The two 
experimental programs were very interested , in meeting the Observers 
before ^hfey entered the classrooms to do observation^ For these two 
programs, meetings were set up for Sunday evening, the day the Site 
Monitor and Observers *were to arrive- on site. For the two Control pro- ' 
grains, entrance meetings were arranged for the first day of data collect 
tion (Monday). For the In-Depth - Control program, meetinMi'toere arranged 
with "tft^Exe cut ive Director as well as with the Head Start* Director. The 
Site Monitor of the £wo Impact-only*programs was also able to arrange 
interview appointments with the teachers (and also Head Start staff, 
for Process-only interviews) . , 



Site Monitors— ^lso^assessed the availability of Interviewers » 
and, wheo possible^h^r^d them. At one site, it was clear- that additional 
effort would be necse#sary to find Interviewers. The Site -Monitors also 
assembled class schedules^ naifies of teacher aides, and made roufeh deter- 
minations of 'the number oi Spanish translations of measures .that would t 
be needed. , * 

¥ f 

* r 

Upon returning from the site development visits, the SUte Moni- 
fcors met with the Research Scientist in charge of the Impact and In-Depth 
evaluation for a debriefing session. • During this seipion, the Researdh 
Scientist was receptive to impressions, different perspectives, suggest- 
ions as well as objective information, sffte Monitors were able to speak 
tp'problems of identifying, and selecting interviewers and obtaining 
permission of parents. In addition, the Site Monitors related effective 
strategies of gaining the cooperation of staff as well- as helping to, 
dissipate anxiety associated with having outside personnel evaluating 
programs. f 

* m » 
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Table 1. Observations and Instruments Completed During Site Visits ^ 



Site 



Georgetown, 
Indiana 
Hillsboro 
Monroe 



TX 



CFHM 
Super. 



Hd.St . 
> Dir. 



MH/CFMH 
Provider 



Coordinator 



Teacher 
CIRCUS 



10 
4 
4 
5 



Hi-Sc. 
Ko/PAI 



45 
26 
32 
26 



Kohn 
Teacher 
& Aides 



*65 
45 
80 
45 



Brovrt 



34 



**40 



Preacott 
SRI-Pr. 
sc .QV.sy. 



35 47 



20 
*42 



2U 



SRI Pr. 
sc . ob . 



28 

K 

29 



Other 

H.S. 

Staff 



# Chid 
v comp 

data 



26 
• 1 
30 
10 



I Chid 
w/one 
ob mlsg 
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fOTAl 



268 
144 
269 

j 



9 



TOTALS 



23 



129 235 



n I .-.V 1 ■ 



141 



4. 67 24 



/*This figure reflects the instruments in the files ss of 6/25/80. However, the log sheet -aho^fil-, #5593 and 5593/9*03 seea to be 



nlssing. * 
**Forty instruments were received, one is missing out of Ihe files as of 6/25. 
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Following site development Visits, letters were sent to all 
Program Directors expressing appreciation for theirs cooperation and 
reminding thera of the dates for upcoming data collection site visits. A 
subsequent letter was sent to announce the* upcoming observation and inter- 
view schedules, / 
* % * * 

DatTa Collection 



.1 



By at least the second day of the site visit, Classroom Obsei 
ers, Teacher Observers, and Interviewers were dispersed to begin the 
data collection tasks. In addition to the child ahd teacher observations, 
Observers administered the Kohn Social Competence Scale to teachers. 
TJie Interviewers had the sole responsibility .for the parent interviews. 

d the[r< 



pie Site Monitor conducted the (record reviews and interviewed the Head 

\ 

Start staff and other key staff. 

Table 1 reflects the number of observations 'and number of 
instruments completed during the site visits. In addition to impact and 
in-depth data, 'process data wafi collected from four Head Start "directors , 
seven MH/CFMH providers, two coordinatofs and one CFMH supervisor. The 
use of proces? instruments for such limited data collection activities 
did not violate OMS regulations. 
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Data Management 



The data management" task began with the packing and shipping 

* 

of completed instruments to the Urban Institute corporate .-of f ices. ,The 
^ite Monittyr had the responsibility of shipping the data to the corporate^ 
office before leaving the site. Upon arrival at the Urban Institute for 
Human Services, the contents of each shipment were- logged. The logging 
procedure 1 was undertaken in two steps. An initial count- sheefc <see Appen- 
dix F) was used to record the number of each item and the 4ar4-*^eceived . 
Following this step, all data were piled in locked" file cabinets. 

* 

1 The second step of logging was performed separately for the 
observation and interview- booklets. The interview booklets (teacher, v 
parent, teacher- Kohns .and Browns) were recorded individually on Impact 
and In-Dep*h Individual Interview Log-in sheets. The Respondent /Inter- 
viewer ID number was ^recorded, as well as the Child ID number in the case 
of Teacher Kohns; the presence of a consent f?rm ^as recorded; the number 
of log and edit sheets was recorded; if the interview had been in Spanish, 
that was recorded. As e^fch booklet was recq*Wed, the ID #'s were checked 
against the Data Collection Master Schedules used by the Site Monitors; 
the consent form wa§^ removed after the R/l ID number was recorded' on it 
and a C.F. notation Was made on the booklet cover; the booklet was flip- 
ped through to. cheese for completeness and incomplete was marked on the 
cover where necessary/ 

, On the second step log- in of Fjrescott and StfF Observation Data 

sheets, the following ihformation was 'recorded : the focus person ID num- 
ber, after checking against the tfasjer Data Collection Schedule; the 
Observer ID number; the booklet numbers, the number of log sheets; and 
fox the Prescotts; the presence of a permission-to-observe slip. 

• » 
A final edit was performed in each of the 262 SRI-Prescott and 
SRI Preschool observation booklets in preparation for having the optically 
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seating*. T^e Observers were retained to complete approximately 80 , 
hours of editing. The editing tasks included: 

(a) checking the numbers written in all ID grids on the 
/ , coyer and throughout the booklet; 

(b) checking thp sequence numbers of the observations; 

(c) locating and cancelling blank, incomplete and illegal 
^ frames; 

(d) checking the size and darkness of each bible; and . , 

(e) checking the log sheets for problems and removing 
•the problem sheets. As well, supplementing identifying 

information was added to the booklets in order to be 
included on the tape of the scanned booklets. This 
information included the' iden£ication numbers of the f 
u - # second observer in the Prescott booklets^ the Head Start 
program site number and the center/classroom identifica- 
tion number. 

The edited booklets were sent to Intrans for optical scanning 
and conversion to magnetic tape, .Ae raw frequency data were then shipped 
to Abt Associates in Cambridge/Mass, for the first stage of data analy- 
sis. At Abt Associates, the frequency data from the SRI Prescott were 
transformed into 3 set of variables t adapted from the National Day Care 
Study and another set of variables developed by the Urban Instituted 
research staff. Another set of variables, including sequence variables, 
were developed Nf torn the SRI -Preschool frequency data*. The second stage 
sis is in progress. 



